Skip to content

Add black and whitelists #83

Merged
merged 18 commits into from
Apr 15, 2020
Merged

Add black and whitelists #83

merged 18 commits into from
Apr 15, 2020

Conversation

donald
Copy link
Contributor

@donald donald commented Apr 15, 2020

Add --blacklist and --whitelist to mxqsub and add mxset command. Replaces #81 and #82

Avoid unneeded warning when scanning JOB_TMPDIR_MNTDIR for possible
leftover job mounts. The parent directory of JOB_TMPDIR_MNTDIR is
created when the first job with the --tmpdir feature is started. So
it won't exist in a new server.

This cleanup path is only used in the exceptional case that the usual
cleanup path (via job_has_finished) didn't succeed, e.g. when mxqd was
killed. The cleanup is not essential for the currently running server.
Add function attribute ((unusged)) to avoid a compiler warning when the
static inline function defined in the header file is not used by the
compilation unit.
Add header file for the quasi-standard [1] xmalloc call.

The current approach implemented in mx_util is to wait and retry on
ENOMEM malloc failure. However, this overhead doesn't seem to be
justified, because it is difficult to imagine a case, where a malloc
would fail with ENOMEM at one time and a retry would succeed.

[1] https://www.gnu.org/software/libc/manual/html_node/Malloc-Examples.html
Add a utility module which can store a set of keywords. The set can be
created and updated from a string and can be serialized into a string.

The canonical string representation produced by this utilits is the space
separated list of the sorted keywords in lexical order.

Input strings used to create or update a keyword set contain keywords
separated by whitespace. The keywords may be prefxied with "+" and "-".
For updates, first all unprefixed keywords are processes to create the
initial set, than updates are applied from the prefixed keywords.

An empty string used for updates is considered "no change", not
"set to empty set".

Usage example:

    struct keywordset *kws = keywordset_new("xx yy")
    keywordset_update(kws, "-yy +zz")  // remove yy, add zz
    keywordset_update(kws, "aa bb")    // set to aa, bb
    if (keywordset_ismember(kws,"bb")) ... // true
    char *s = keywordset_get(kws):   // s now "aa bb"
    free(s);                         // caller must free()
    keywordset_purge(kws)            // set nowto empty.
    keywordset_free(kws);
Add list of blacklisted and whitelisted servers to group. The string is
supposed to be canonical (lexical sorted, space separated) as produced
by keywordset_get(), so that groups with equal sets can be found by sql
string compare.
Add list of blacklisted and whitelisted servers to group. The string is
supposed to be canonical (lexical sorted, space separated) as produced
by keywordset_get(), so that groups with equal sets can be found by sql
string compare.
Add a new options so that specific mxqd servers can be excluded from starting
jobs for this group.

    mxqsub --whitelist "acedia avaritia" sleep 10   // start only on these
    mxqsub --blacklist dontpanic sleep 10 // start on any node but this one

Blacklist has priority, so

    mxqsub --white "kronos, uselessbox" --black kronos

would not start on kronos. The lists of blacklisted and whitelisted
servers can be modified with mxqset.
Add a new command which can be used to modify an existing group.

Examples:

    mxqset group 123 --closed
    mxqset group 123 --open
    mxqset group 123 --blacklist "dontpanic"          # replace
    mxqset group 123 --whitelist "+uselessbox +gula"  # add
    mxqset group 123 --blacklist -dontpanic           # remove
    mxqset group 123 --whitelist ""                   # clear

The flags open and closed can be set from mxqadmin as well
(`mxqadmin --close=123`) , but the synopsis, in which the options take
the groupid as a value is difficult to expand to new options, which take a
value.
Check, whether this server is qualified to start jobs from a group.

Lazy-evaluate qualification criteria defined in the group and cache the result.

The qualification criteria need to be reevaluated  when the active groups
were reloaded, because they may be changed by the user in existing
groups.

For now, the only qualification criteria are the groups blacklist and
whitelist. If a group has a whitelist, the short or long hostname of the
mxqd server needs to be on that list, otherwise the server is not
qualified for the group.

If the servers name is on the blacklist of the group, the server is not
qualified for the group.

Don't start jobs we are not qualified for.

This can later be expanded to additional criteria (e.g. hostconfig or
processor flags).
@arndt
Copy link
Contributor

arndt commented Apr 15, 2020

I can't wait to see this in action.

@donald donald merged commit ac79166 into master Apr 15, 2020
@donald
Copy link
Contributor Author

donald commented Apr 15, 2020

Go ahead, its running on all servers now.

mxqsub --whitelist acedia sleep 10

was executed ... on ... acedia :-)

http://afk.molgen.mpg.de/mxq/mxq/job/20714836

@arndt
Copy link
Contributor

arndt commented Apr 15, 2020

ok - I just blacklisted 'dontpanic' and did not need to panic anymore. Hurray!

@arndt
Copy link
Contributor

arndt commented Apr 15, 2020

... wait a sec ... - I need more testing - just saw a job crashing on dontpanic -

http://afk.molgen.mpg.de/mxq/mxq/job/20716429

Did I do something wrong?

@arndt
Copy link
Contributor

arndt commented Apr 15, 2020

probably the dashes '? How do I have to format a list with one host and with multiple hosts?

@donald
Copy link
Contributor Author

donald commented Apr 15, 2020

Yes, the dashes quotes. I wonder, how you managed to get them in there....

mxqsub --blacklist dontpanic
mxqsub --blacklist dontpanic
mxqsub --blacklist "dontpanic"
mxqsub --blacklist 'dontpanic'

should all do. All the same. Witha list (spaces) you'd need some kind of quotes. And

mxqsub --blacklist 'dontpanic'

is the same as

mxqsub --blacklist='dontpanic'

@donald
Copy link
Contributor Author

donald commented Apr 15, 2020

Or even

mxqsub --blac"klist=don"tpa'n'i\c

@arndt
Copy link
Contributor

arndt commented Apr 15, 2020

ok - klappt

@donald donald deleted the add-blackwhitelists branch October 28, 2022 14:22
Sign in to join this conversation on GitHub.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants